Evolution of direct reciprocity in group-structured populations

People tend to have their social interactions with members of their own community. Such group-structured interactions can have a profound impact on the behaviors that evolve. Group structure affects the way people cooperate, and how they reciprocate each other’s cooperative actions. Past work has shown that population structure and reciprocity can both promote the evolution of cooperation. Yet the impact of these mechanisms has been typically studied in isolation. In this work, we study how the two mechanisms interact. Using a game-theoretic model, we explore how people engage in reciprocal cooperation in group-structured populations, compared to well-mixed populations of equal size. In this model, the population is subdivided into groups. Individuals engage in pairwise interactions within groups while they also have chances to imitate strategies outside the groups. To derive analytical results, we focus on two scenarios. In the first scenario, we assume a complete separation of time scales. Mutations are rare compared to between-group comparisons, which themselves are rare compared to within-group comparisons. In the second scenario, there is a partial separation of time scales, where mutations and between-group comparisons occur at a comparable rate. In both scenarios, we find that the effect of population structure depends on the benefit of cooperation. When this benefit is small, group-structured populations are more cooperative. But when the benefit is large, well-mixed populations result in more cooperation. Overall, our results reveal how group structure can sometimes enhance and sometimes suppress the evolution of cooperation.

. A schematic representation of the model setting. (a) We consider pairwise interactions in groupstructured populations. The population is composed of M groups (depicted by grey sets). Each group contains N players (depicted by colored circles). In this example, M = 4 and N = 3 , such that the total population size is NM = 12 . The colors of the small circles indicate the players' strategies. The black arrows between these circles indicate that each player interacts in a repeated prisoner's dilemma with all other group members. Over time, players may change their strategies for the repeated prisoner's dilemma in three different ways. (b) First, with probability µ in , players engage in intra-group imitation. In that case, a player randomly samples a role model from the same group, and imitates the role model's strategy with a certain probability that depends on the players' payoffs. (c) Second, with probability µ out , players engage in inter-group (or out-group) imitation. In that case, the player randomly selects a role model from a different group, and again imitates this role model with a certain probability. (d) Finally, with probability ν , there is a mutation, in which case the player selects to a new strategy randomly. The three probabilities sum up to one, µ in + µ out + ν = 1 . When we consider a scenario with a complete separation of time scales, we assume ν ≪ µ out ≪ µ in . When we consider a partial separation of time scales, we assume ν ≪ µ in and µ out ≪ µ in .

Model
The following description of our model consists of two parts. First, we describe how individuals engage in pairwise interactions within their groups. These interactions take the form of an infinitely repeated prisoner's dilemma. We derive the payoffs that each player obtains, given the players' strategies. In a second step, we describe how individuals update their strategies over time, based on a pairwise comparison process 34 . Dynamics of the repeated prisoner's dilemma. The repeated Prisoner's Dilemma (PD) is the most fundamental theoretical framework to study direct reciprocity. The game takes place among two players. In each of infinitely many rounds, each player independently decides whether to cooperate (C) or to defect (D). In this paper, we study a special variant of the prisoner's dilemma, the donation game. Here, cooperation means that a player pays a cost (normalized to c = 1 ) in order for the other player to receive a benefit b > 1 . The respective payoff matrix is If the donation game is only played once, the only Nash equilibrium is to mutually defect. But in the repeated game it can be reasonable to cooperate, because players may get a higher long run payoff by maintaining a good relationship with their co-player. In particular, the Folk theorem of repeated games guarantees that mutual cooperation can be sustained in a Nash equilibrium, provided there are sufficiently many rounds 35 .
In general, strategies for the repeated prisoner's dilemma need to tell the player what to do after any history of previous interactions. The resulting space of possible strategies is vast 36 . To simplify our analysis, in the following we focus on a well-known subspace, the space of memory-1 strategies 31 . When players adopt a memory-1 strategy, they only condition their next decision on what happened in the previous round. Such strategies can be represented as a 4-tuple, An entry p ij represents the player's cooperation probability, given that in the previous round the player used action i and the co-player used action j. We call a memory-1 strategy deterministic, if all entries are either zero or one, and we refer to the set of all such strategies as M . It follows that there are |M| = 2 4 = 16 such strategies in total. They are summarized in Table 1. In particular, this table includes several well-known strategies such as AllD, Tit-for-Tat (TFT), or Win-Stay-Lose-Shift (WSLS) 10,11 . For our subsequent analysis, we shall assume that players make errors with some small probability e. That is, with probability e a player defects although this player (2) p = (p CC , p CD , p DC , p DD ). Table 1. Deterministic memory-1 strategies of the repeated prisoner's dilemma. Memory-1 strategies only depend on the outcome of the previous round. These strategies are deterministic if, given the previous outcome, they either prescribe to cooperate with certainty (C), or to defect with certainty (D). The four letters in the second column represent which action the strategy prescribes, given the previous actions of the focal player (first letter) and the co-player (second letter). The third column shows the cooperation level when both players use the respective strategy γ p,p (assuming e → 0 ). See "Methods" for details of the calculation. The fourth column indicates whether or not the strategy is a rival strategy. A strategy p is a rival if it enforces the payoff relationship π p,q ≥ π q,p against all co-player's strategies q . The fifth column shows how each strategy p performs against WSLS (again for e → 0 ). If π(p, WSLS) > π(WSLS, p) , then we mark the strategy p with a ' + ' . Analogously, if payoffs are equal, we use a '0' , and if the payoff is smaller, we use '−' . Provided the benefit b > 1 , these relationships are independent of the exact value of b.

ID
Prescriptions (CC, CD, DC, DD) Self-cooperation level Rival vs WSLS www.nature.com/scientificreports/ intended to cooperate (and conversely, a player who intended to defect may cooperate). As a result, instead of their intended strategies p , players implement the effective strategies (1−e)p + e(1 − p). When both players adopt a memory-1 strategy, one can explicitly compute their payoffs and how often they cooperate, by representing the game as a Markov chain 6 . The possible states of this Markov chain are the possible outcomes of each round (CC, CD, DC, DD; here, the first and second letter refer to the action of the first and second player, respectively). Given these states and the players' (effective) strategies p = (p CC , p CD , p DC , p DD ) and q = (q CC , q CD , q DC , q DD ) , the transition matrix T of the Markov chain takes the form Here, p ij = 1 − p ij and q ij = 1 − q ij for i, j ∈{C, D} are the players' respective defection probabilities. For positive error rates, every entry of this transition matrix is positive. It follows by the Theorem of Perron-Frobenius that T has a unique invariant distribution v = (v CC , v CD , v DC , v DD ) . In particular, the p-player's average cooperation level is γ p,q := v CC +v CD whereas the q-player's cooperation level is γ q,p := v CC +v DC . As a result, the p-player's long-term average payoff is given by When both players adopt the same strategy p = q and errors are rare, e → 0 , Table 1 shows the cooperation levels against itself γ p,p for each deterministic memory-1 strategy. In particular, the table reveals that there are only three deterministic strategies that fully cooperate against themselves in the limit of rare errors. These strategies are AllC, WSLS, and the strategy p = (1, 1, 1, 0) . Out of these three strategies, WSLS is a Nash equilibrium if b ≥ 2c , whereas the other two strategies are always unstable 6 .
Evolutionary dynamics in group-structured populations.. To study the evolution of reciprocity in group-structured populations, we extend the models of Hauert, Chen, and Imhof 8,9 (who introduced the formalism for one-shot, non-repeated games). We consider a finite population of size MN subdivided into M groups of size N. Individuals in each group engage in pairwise interactions with all other members of their group, as depicted in Fig. 1a.
To compute the overall payoff of a focal individual with strategy p , let n q denote the number of group members that adopt strategy q ∈ M . Because the p-player interacts with all N −1 other group members, the player's average payoff is Here, π p,q is the payoff that the p-player obtains against a co-player with strategy q , as defined by Eq. (4), and δ p,q is the indicator function that is one if p = q , and zero otherwise.
The players' strategies are not fixed. Instead, each player updates its strategy according to the following dynamics. At each time step, one player from the population is chosen randomly as the focal player. Suppose this player currently uses strategy p . The focal player is then given a chance to adapt its strategy, either by intra-group imitation (with probability µ in ), out-group imitation (with probability µ out ), or mutation (with probability ν ), as depicted in Fig. 1b-d. In particular, µ in + µ out + ν = 1 . In case of an intra-group imitation,the focal player randomly selects a role model from its own group. If the role model adopts strategy q , the focal player switches to the role model's strategy with a probability given by the Fermi function 37,38 Here, σ in ≥ 0 represents the selection strength of intra-group imitation. If σ in is small, imitation events are mostly driven by chance, f in p→q ≈ 1/2 . In contrast, if σ in is large, the role model's strategy only has a reasonable chance of being imitated if it yields at least the payoff of the focal player.
The case of out-group imitation follows an analogous procedure. Here, the focal player randomly selects a role model from a different group (with all other groups being equally likely). If the respective role model happens to use strategy q , the focal player adopts this strategy with probability Here, σ out is the selection strength for out-group imitation. Note that because the focal player and the role model are now in different groups, they do not play the game with each other, which is one of the key differences from models without group structure. Out-group imitation plays a similar role as migrations in genetic models of evolution 8,12 . It allows strategies to move from one group to another. Finally, in case the focal player changes its strategy by mutation, the player simply replaces its current strategy p by a random strategy q . All deterministic memory-1 strategies q are equally likely to be chosen.
The above elementary updating process is iterated for many time steps. In each time step, a single individual is given the chance to update its strategy by intra-group imitation, out-group imitation, or mutation. Overall, (7) f out p→q = 1 1 + exp σ out π p − π q . www.nature.com/scientificreports/ this gives rise to a stochastic process on the space of all population compositions. In contrast to typical multilevel selection models 12,39 , where an entire group may be replaced by another group, selection always operates on the individual level. Successful groups do not replace less successful ones; rather strategies of successful players are more likely to be imitated over time. The resulting stochastic process is straightforward to simulate. In the following, we derive results for two important special cases. In the first special case, we assume a complete separation of time-scales. Here, mutations are rare compared to out-group comparisons, which themselves are rare compared to intra-group comparisons, ν ≪ µ out ≪ µ in . In the second case, we assume that mutations and out-group comparisons happen on a similar time scale, but both are rare compared to intra-group comparisons, ν ≪ µ in and µ out ≪ µ in .

Analysis
Dynamics when there is a full separation of time scales. We begin by assuming a complete separation of time scales, ν ≪ µ out ≪ µ in . In this setting, the intra-group dynamics are fast compared to the others. As a result, at any point in time there are at most two different strategies present in any group. When a mutation or an out-group imitation introduces a new strategy, intra-group imitation leads to the extinction or fixation of this strategy before the next strategy is introduced. In the following, we describe this dynamics in more detail.
Description of the intra-group dynamics. Consider a group in which i players adopt the strategy p and N −i players adopt the strategy q . By Eq. (5), the players' payoffs are given by (see "Methods" for an alternative formulation) The probability that intra-group imitation increases (decreases) the number of p-players in a single time step is Here, the first equation corresponds to the case where a q-player is randomly chosen as the updating player, a p-player is chosen as the role model, and the updating player chooses to imitate the role model. The second equation corresponds to the converse case of a p-player imitating a role model with strategy q.
The fixation probability of a single p player in a resident group of q-players can be computed explicitly 40,41 . This probability is given by ρ p,q = 1 . By using the explicit payoff equations (8), this fixation probability becomes 22 Using this formula, we can compute for each resident strategy q how likely it is that any novel strategy p is eventually adopted by the entire group. While the use of fixation probabilities has become common practice in evolutionary game theory 41 , we note that the time it takes for a single strategy to reach fixation may be considerable. The fixation time becomes particularly long when groups are large, and when the strategies p and q allow for an equilibrium in which the two strategies stably co-exist 42 . Nevertheless, this limit has become a useful approximation, as it simplifies computations considerably. Instead of considering arbitrarily many strategies at once, one can make predictions by only considering two strategies at a time 43,44 . Once a strict separation of time scales does no longer apply, the analysis becomes considerably more intricate 45 .
Description of the inter-group dynamics.. To further simplify the analysis of our model, we make the additional assumption that ν ≪ µ out . This limit indicates that the time scale for out-group imitations is short compared to the time scale of mutations. This assumption implies that at any point in time, at most two different strategies are present in the entire population. Once a mutation introduces a new strategy, this strategy either fixes in the population (through successive in-group and out-group imitation events), or the strategy goes extinct. To describe this dynamics in more detail, suppose that the two strategies p and q are present in the population. Since intra-group imitation is fast, every group is homogeneous. As a consequence, we can speak of p-groups and q -groups, depending on which strategy the group members employ. Once a q-player imitates a player from a p -group, the number of p-groups may increase (if the strategy p reaches fixation in the q-group). The respective probability that the number i of p-groups increases (or decreases) is given by www.nature.com/scientificreports/ respectively. In both expressions, the first three factors on the right hand side represent the probability of the respective out-group imitation event. The last factor is the probability that the newly introduced strategy reaches fixation. The ratio η of these transition probabilities simplifies to From Eq. (11), the ratio of the intra-group fixation probabilities is 46 Thus, Eq. (14) can be re-written as Overall, we obtain the following formula for the probability that a new strategy p takes over the entire population, given everyone else applies strategy q, Here, the first factor ρ p,q is the probability that the p-mutant takes over the first group. The second factor gives the probability that eventually also all other groups adopt p . Similarly one can calculate the fixation probability that everyone in a p-population eventually imitates a single q mutant. This probability is These formula allow us to compute how likely any given mutant strategy is to replace the resident strategy when there is a complete separation of time scales.
Strategies favored by the evolutionary process. Using these formulas, we can analyze which strategies are particularly likely to spread. To this end, we say strategy p is favored over q if a single p-mutant is more likely to fix in a q-population than vice versa. By Eqs. (17) and (18), the respective condition � p,q > � q,p simplifies to The left hand side reflects the effect of in-group imitation, whereas the right hand side captures the effect of out-group imitation. In the special case of a single group, M = 1 , this condition reproduces the respective condition for well-mixed populations, ρ p,q > ρ q,p . Plugging Eq. (15) into Eq. (19) yields By collecting alike terms, this expression can be further simplified to The first and the second terms of this inequality correspond to the dynamics within and between groups, respectively. The intra-group dynamics is decisive if either σ in N ≫ σ out or if the number of groups is small ( M ≈ 1 ). In that case, and if groups are additionally assumed to be small ( N → 2 ), the condition for p to be favored simplifies to This condition is closely related to the notion of rival strategies 14 . Strategy p is a rival strategy if and only if it enforces the condition π p,q ≥ π q,p against all strategies q . In Table 1, the second-to-last column indicates all memory-1 rival strategies. There are four of them, TFT, Grim, AllD, and the strategy p = (0, 0, 1, 0) . The above observations suggest that these rival strategies should be particularly strong when there is only a single group with two group members ( M = 1 and N = 2).
In the other extreme, when σ in N ≪ σ out and M is sufficiently large, it is the inter-group dynamics that is decisive. In that case, the relative strength of a strategy is determined by its efficiency. Strategy p is favored over q if and only if it yields the larger payoff against itself,  47 , independent of the exact values of M and N. Equation (24) indicates that the preference between two strategies is independent of N and M as long as the total population size MN is fixed. We note, however, that the extent to which p is preferred over q does depend on M and N. Thus, the relative abundance of the strategies changes according to the group structure when there are more than two strategies, as we will see in the following numerical simulations.
Numerical simulations. The above arguments are valid only when players choose among two strategies. In the following, we explore evolution among all 16 deterministic memory-1 strategies by implementing the evolutionary process numerically. To this end, we use Monte Carlo simulations. Mutant strategies are repeatedly introduced into the current resident population. The mutant strategy either takes over or goes extinct. We report how much cooperation we observe on average (see ""Methods"). Figure 2 shows how the evolving cooperation level depends on the number of groups M, either for small groups ( N = 2 ) or for relatively large groups ( N = 32 ). When the group size is small, we observe very little cooperation if there is only a single group ( M = 1 ), as predicted by our earlier analysis. As we increase the number of groups, also the cooperation level increases. However, they do not improve indefinitely. Rather, these improvements saturate as we increase M, which is consistent with the factor (M − 1)/M in Eq. (20). The limiting cooperation level depends on the benefit of cooperation, reproducing the standard result that larger benefits are more conducive to cooperation 7 . In general, we thus observe that cooperation tends to be favored when M, N, and b are large, corresponding to many groups of substantial size, and a considerable benefit to cooperation.
After exploring the effect of group size and number of groups in isolation, we next ask to which extent group structure facilitates cooperation. To this end, we keep the total population size fixed at MN = 120 , and vary the group size N. The number of groups is then automatically determined as M = 120/N . In one extreme case, there is only a single group of maximum size, M = 1 and N = 120 . We refer to this scenario as the case of a (single) well-mixed population. In the other extreme case, groups take the minimum non-trivial size, N = 2 , which implies that the resulting number of groups is M = 60 . We refer to this second scenario as the case of a (fully) group-structured population. Figure 3a shows how the cooperation level changes as we vary the group size N. Interestingly, the effect of group structure depends on the benefit b of cooperation. If the benefit is small, groupstructured populations achieve more cooperation than well-mixed populations. For intermediate benefits, we observe the opposite trend. Here, well-mixed populations are more conducive to cooperation. Finally, once benefits are very large, full cooperation evolves in all considered cases, independent of the exact values of M and N.
To further investigate these non-trivial effects of group structure, we analyze the abundance of each of the 16 strategies in the stationary state, see Fig. 3b-e. We first consider the case that the benefit of cooperation is intermediate, b = 3 . Here, well-mixed populations lead to much more cooperation. In particular, here we observe that populations learn to adopt the cooperative Win-Stay Lose-Shift (WSLS) strategy almost all of the time (Fig. 3c). In group-structured populations, on the other hand, no single strategy is predominant. The most abundant strategies are the non-cooperative strategies AllD and Grim. The next abundant strategies are WSLS and TFT, respectively. Overall, we thus observe that well-mixed populations are more favorable for cooperation (23) π p,p > π q,q .
(25) π p,p + π p,q − π q,p − π q,q > 0. www.nature.com/scientificreports/ because they make it more likely that the cooperative strategy WSLS evolves. In a second step, we consider the case of low cooperation benefits, see Fig. 3d,e. Group-structured populations again lead to the evolution of the strategies AllD, Grim, TFT, and WSLS (and related strategies). In contrast, well-mixed populations consist of the non-cooperative strategies AllD and Grim almost entirely.
To better understand why cooperative strategies are abundant in one scenario but not in another, we investigate the transition probabilities for a reduced strategy space. The reduced strategy space contains the representative strategies AllC, WSLS, TFT, AllD and the strategy S 7 with p = (0, 0, 0, 1) . Strategy S 7 is included because it has the highest ability to invade WSLS among the memory-1 strategies. Strategy S 7 is preferred over WSLS for the broadest range of b according to Eq. (25). The payoffs and the win-lose relationships of these strategies are summarized in Tables 2 and 3. In addition, Fig. 4 illustrates the average abundance of each of these five strategies and the transition probabilities between them. We confirmed that the overall cooperation levels for this reduced strategy space are comparable to the corresponding results for the full strategy space. Hence, the reduced strategy space can serve a useful proxy to gain insights into the overall dynamics.
We first consider the case of an intermediate benefit, b = 3 . Here, well-mixed populations yield more cooperation, as they promote the evolution of WSLS. Figure 4b shows why. There is an evolutionary path from every other strategy towards WSLS; once the entire population adopts WSLS, every other mutant strategy is at a disadvantage. This picture is in line with previous research on direct reciprocity in well-mixed populations 31 . The picture changes, however, in group-structured populations, see Fig. 4a depicting the case of groups of size N = 2 . Here, a homogeneous WSLS population can be invaded by S 7 . To see why, consider a group that contains both strategies. By Table 2, the payoff of WSLS is 1/3b − 2/3 , which is below the payoff of S 7 , 2/3b − 1/3 . Hence, S 7 is favored in each mixed group. On the other hand, with respect to out-group imitation, it is WSLS that is favored over S 7 , because the payoff of WSLS against itself is b−1 , which exceeds S 7 's self-payoff of (b − 1)/2 . To compute which of the two opposing effects dominates, Eq. (21) suggests that we need to compute the sign of π p,p + π p,q − π q,p − π q,q . For b < 5 , this criterion suggests that S 7 is favored (as also indicated in Table 3). These observations explain why in group-structured populations, WSLS is susceptible to invasion by S 7 , which in turn can be invaded by AllD.
In a next step, we explore the case of a small benefit of cooperation, b = 1.5 . Here, group-structured populations are more cooperative. The respective transition graphs for group-structured and well-mixed populations are depicted in Fig. 4c, d. In both cases, we observe that there is no single strategy that resists invasion by all other strategies. Instead, AllD populations are susceptible to TFT, which in turn is susceptible to AllC and WSLS, which can be invaded by AllD again. The main difference between group-structured and well-mixed populations is the relative performance of TFT. Compared to well-mixed populations, TFT is better able to invade AllD populations in structured populations. To see why, we first consider the within-group dynamics when N = 2 . Because TFT gets the same payoff as the opponent in any pairwise encounter 15 , the fixation probability of TFT in a group with ALLD is exactly 1/2. In addition, TFT is favored by the between-group dynamics, because the payoff of TFT-groups is (b − c)/2, which is larger than the payoff of zero in AllD-groups. It follows that a single TFT mutant can replace an AllD population with a probability that is approximately 1/2. In contrast, in wellmixed populations, this fixation probability is much smaller, 0.18 for the parameters in Fig. 4d.
The above results suggest that overall, there are two competing effects when splitting a population into smaller groups. On the one hand, smaller group sizes favor the evolution of rival strategies because small groups generally   Table 2. Payoff matrix for the reduced strategy space, consisting of AllC, WSLS, S 7 , TFT, and AllD. The entry (i, j) is the payoff π ij that the (row) strategy i obtains against the (column) strategy j. Table 3. Pairwise comparisons for the reduced strategy space. Again, we consider every pairwise combination of AllC, WSLS, S 7 , TFT, and AllD. The cells indicate the sign of π p,p + π p,q − π q,p − π q,q , the left-hand side of Eq. (25). A plus sign (minus sign) indicates that the row strategy p is favored (disfavored) over the column strategy q . If the sign depends on the benefit b, the range for which the sign is positive is shown. Blank cells indicate that the left-hand side of Eq. (25) is equal to zero. www.nature.com/scientificreports/ select for spite 40 . On the other hand, group structure can favor the evolution of cooperation because individuals in highly cooperative groups are more likely imitated. Our above results suggest that the overall outcome of these two opposing effects depends on the benefit of cooperation. When this benefit is comparably small, group-structured populations allow for more cooperation than well-mixed populations. In contrast, when this benefit is intermediate, cooperation in well-mixed populations is more robust.
Dynamics when there is a partial separation of time scales. Throughout our analysis so far we have assumed a complete separation of time-scales. When a player was randomly chosen to update its strategy, we assumed that this player is most likely to engage in intra-group imitation, far less likely to engage in out-group imitation, and again far less likely in random exploration (mutation). In the following, we instead assume that intra-group comparisons are still most likely; however, mutations and out-group comparisons now occur on a similar time scale. In this limit, all groups can be assumed to be homogeneous because intra-group imitation is fast. However, different groups might employ different strategies, because mutations might introduce novel strategies faster than out-group imitation can result in the fixation of any given strategy in the population.
A differential equation in the limit of large populations. To obtain analytical results, in the following we assume that the number of groups is large, M → ∞ . Let x p be the fraction of groups that employ strategy p . Over time, these fractions can change, either because new strategies are introduced into groups by out-group imitation (and reach fixation), or they are introduced by mutations (and reach fixation). This dynamics may be described by the following differential equation, Here, r = ν/(ν + µ out ) is the relative mutation probability (compared to out-group imitation events). The right hand side of Eq. (26) consists of two parts. The first sum describes the changes triggered by out-group imitation. Here, the parameter describe the flow from strategy q to strategy p . For example, the denominator of the first term on the right hand side describes the likelihood that a q-player switches to p due to out-group imitation. The numerator describes the likelihood that subsequently, p reaches fixation due to in-group imitation. The interpretation of the second term on the right hand side is similar, by considering the possibility that a p-group makes the converse transition towards q . The second sum in Eq. (26) describes the changes triggered by mutation events. Here, the denominator of ρ p,q /|M| describes the probability that the mutating player adopts strategy p . The numerator gives the probability that this strategy is then adopted by all other group members, due to intra-group imitation. We note that the sum p x p = 1 by definition, and hence the equation is defined on the 16-dimensional simplex. Moreover, since pẋ p = 0 , the unit simplex is invariant under the dynamics. One may interpret Eq. (26) as a variant of the replicator-mutator equation 48 , where the first part represents selection and the second part represents mutations.
Further below, we explore the solutions of Eq. (26) numerically, for various parameter combinations. For all parameters we considered, the dynamics converges to a stable fixed point. Such a fixed point satisfies the equation We would like to emphasize that the Eq. (26) does not need to recover the qualitative dynamics that we obtained in the previous section, even when r → 0 (in which case mutations are again rare compared to outgroup imitation events). In other words, the order in which limits are taken affects the solution that is predicted. As we show further below, however, the solutions predicted by Eq. (26) are in excellent agreement with explicit simulations of the evolutionary process for all values of r we considered.
Numerical results. Figure 5a shows the evolving cooperation levels for a well-mixed population ( N = 200, M = 1 ) and a group-structured population ( N = 2 , M = 100 ). We observe a striking difference between the two settings. In the well-mixed population, the cooperation level strongly depends on the benefit of cooperation, as one may expect. For small benefit values, hardly any cooperation evolves. For intermediate and large benefit values, almost the entire population cooperates eventually. In contrast, in the group-structured population, cooperation levels are around 1/2 when r is low, largely independent of the benefit b. For r O(10 −1 ) , the cooperation levels drop as r increases, as shown in Fig. 5b.
To explore these results for group-structured populations in more detail, Fig. 6a shows the abundance of strategies in the selection-mutation equilibrium for b = 3 . According to this figure, the most abundant strategy is WSLS, followed by S 7 , Grim, AllD, and S 13 (the latter four strategies are exactly the strategies that have an advantage when directly competing with WSLS, see right-most column of Table 1). The underlying evolutionary dynamics are schematically depicted in Fig. 6b. Individuals in groups with non-cooperative strategies (such as Grim and ALLD) tend to adopt more cooperative strategies like TFT by out-group imitation. Once such groups contain a TFT-player, TFT may reach fixation by intra-group imitation (TFT is neutral when there is only a single TFT player in the group, and it is selectively favored when there are two TFT players or more). TFT-groups in turn are easily replaced by strategies that are more cooperative in the presence of errors, such as AllC and WSLS.
(27) α p,q ≡ ρ p,q 1 + exp σ out π q,q − π p,p − ρ q,p 1 + exp σ out π p,p − π q,q (28) x * p = r · q� =p ρ p,q x * q /|M| (1 − r) q� =p α p,q x * q − r q� =p ρ q,p /|M| www.nature.com/scientificreports/ WSLS groups are comparably stable; as they reach the maximum payoff against themselves, individuals in these groups are unlikely to learn non-cooperative strategies by out-group imitation. However, strategies like AllD and Grim may invade a group of WSLS players once they are introduced by mutations. Assuming that the group is small (the figure depicts the case of N = 2 ), AllD and Grim are both likely to take over, thereby closing the evolutionary cycle. Importantly, the above arguments do not depend on the precise value of b; they only depend on the win-lose relationships between strategies. This argument can thus explain why we observe a coexistence between WSLS and non-cooperative strategies for a wide range of benefit values. The above argument also explains the dependency of cooperation levels on r. When r is sufficiently small, the abundance of TFT also becomes as small as O(r): the transitions between WSLS and the defectors thus balance, and their abundance as well as cooperation levels do not show significant change as r varies. However, because the flow from WSLS to the defectors are mainly driven by the mutation events, the abundance of WSLS players drops as mutation events get more frequent.
When r or M is even smaller, evolutionary results become closer to the results observed when there is a complete separation of time scales. The crossover point depends on the strength of selection. When selection strengths are sufficiently weak, the evolutionary dynamics get closer to neutral selection, where fixation times are relatively short. Here, either WSLS or AllD may happen to take over the entire population with a non-negligible frequency, and the evolutionary dynamics are better described by the complete separation of time scale.  . WSLS is most abundant, followed by S 7 , Grim, AllD, and S 13 . (b) A schematic diagram of the typical transitions between strategies reveals that groups of AllD and Grim players tend to transition towards TFT by out-group imitation. TFT groups in turn tend to transition towards more cooperative strategies like WSLS or AllC. Finally, WSLS groups tend to transition towards AllD and Grim once these strategies are introduced by mutations.

Summary and discussion
Herein, we propose and study a model for the evolutionary dynamics of direct reciprocity in group-structured populations. In our model, individuals can adopt new strategies in three different ways. They may imitate members of their own group (with whom they also engage in prisoner's dilemma interactions); they may imitate members from a different group (with whom they do not interact directly); or they may adopt a new strategy by random strategy exploration (similar to a mutation). While we derive the model in general terms, we focus on two special cases in particular. In the first case, there is.a complete separation of time scales. Here in-group imitation occurs much more often than out-group imitation, which in turn occurs more often than mutations. For this case, we can analytically derive the fixation probability, the conditional and unconditional fixation times (Appendix), and the condition for a strategy to be favored in a pairwise competition. In the second case, there is only a partial separation of time scales. In-group imitation still occurs most often, but now out-group imitation and mutations happen at a similar rate. We explore this case by Monte Carlo simulations, and by analyzing the properties of a differential equations that is valid in the limit as the number of groups becomes large. For both cases, we explore the effect of group structure by comparing the abundance of cooperation for group-structured populations with the corresponding results for well-mixed populations of equal size. Interestingly, we find that the effect of group structure depends on the benefit of cooperation. When this benefit is small, group-structured populations are more cooperative than well-mixed populations. This ordering reverses once the benefit of cooperation is intermediate or large. This result differs from previous models on the evolution of reciprocity in the presence of population structure. In van Veelen et al. 49 , the authors explore the evolution of strategies in repeated games with relatedness. In their model, there is an assortment parameter α that determines how likely players who use the same strategy are to interact with each other. Their simulation results displayed in their Fig. 2 suggest that the effect of this kind of population structure is always positive. As the assortment parameter α increase, people tend to adopt more cooperative strategies on average. In contrast, we find that additional group structure can sometimes prevent the evolution of cooperation. This effect occurs because the effect of group structure is ambivalent in our model. On the one hand, splitting a well-mixed population into smaller groups promotes cooperation, because people in cooperative groups are more likely to act as role models for between-group comparisons. On the other hand, group-structured populations lead to smaller effective population sizes, which in turn select for spite 40 and defection 7 . The overall outcome of these two opposing effects depends on how profitable cooperation is.
To derive our results, we have focused on a comparably simple strategy space, the space of memory-1 strategies 6 . One of the open questions is thus the study of strategies with longer memory. According to Eq. (21), there are two requirements for a strategy p to be successful in group-structured populations. They need to be efficient ( π p,p ≥ π q,q ), but at the same time they should also have a higher payoff than the co-player in a direct interaction ( π p,q ≥ π q,p ). Recent research suggests that strategies exist that satisfy both of these conditions simultaneously. These so-called friendly rivals [50][51][52][53][54] are fully cooperative when they interact with one another, yielding the maximum payoff of π p,p = b − 1 . However, they also make sure they are never outperformed by any opponent, π p,q ≥ π q,p for all co-player's strategies q . It is straightforward to show that when the resident applies a friendly rival strategy p , the fixation probability q,p of any mutant is at most 1/MN. In other words, friendly rivals are evolutionarily robust 22 for any environmental condition N, M, and b. Instantiating a friendly rival strategy, however, requires more than one-round memory [50][51][52][53][54] . Exploring whether these strategies are particularly favored in group-structured populations is thus an interesting and promising research area for future studies.
There are several directions for future research. While this study considers pairwise interactions with all the members in each group, it would be interesting to study a model of N-player public goods games with the group structure instead. There are a couple of previous studies on the public goods games in structured population 33,55 , and we expect further studies in this direction may lead to some universal insights shared between these models. Secondly, we may extend the model so that it allows overlapping membership between groups as the overlapping communities are commonly observed in social networks 56 . Such an overlap may significantly change the model behaviors. Another possible direction is to compare the model with human behavior in empirical or experimental studies. Group-structured populations could better describe human behavior than the more traditional wellmixed population model because human relationships often involve a limited number of people. At the same time, it seems comparably easy in our modern societies to get payoff information even from people that one has no personal ties with. It has long been recognized in sociology 57 , that such weak links between communities may play a pivotal role, which may also affect the evolution of direct reciprocity.

Methods
Cooperation level. In Table 1, we have listed the self-cooperation level for every memory-1 strategy. As has been discussed below Eq. (3), the basic procedure is to find the invariant distribution v = (v CC , v CD , v DC , v DD ) from the principal eigenvector and calculate the cooperation level γ p,q := v CC +v CD . It is known that one can begin by defining 15 where f := (f 1 , f 2 , f 3 , f 4 ) . Let us additionally define f 1 := (1, 1, 1, 1) , f CC := (1, 0, 0, 0) , and f CD := (0, 1, 0, 0) . Then, a closed-form expression for the cooperation level is given as www.nature.com/scientificreports/ When p = q , in particular, we have where X := p CD + p DC , Y := p CD p DC , and Z := p CC p DD . For example, when we calculate the self-cooperation level for S 2 (see Table 1), we can substitute p = (p CC , p CD , p DC , p DD ) = (1 − e, e, 1 − e, 1 − e) into Eq. (31) and take the limit of e → 0 to obtain Payoff transformation. Let us define transformed payoffs as follows 33 : In each of these expressions, the first term means the direct payoff to the focal player, whereas the second term means the co-player's payoff multiplied by −1/(N − 1) , the mean relatedness between a player and its co-player in a well-mixed population of size N 32 . From Eq. (8), we see that This expression shows that the relative advantage of p over q in our two-player game is regarded as an average difference between the transformed payoffs, α k − β k , and the weighting factors in front of them are the respective probabilities of choosing a p-player and a q-player as the co-player from a population of size N − 2.
Alternatively, we may define The idea behind this transformation is the following: for a game with payoff matrix A in a finite population of size N, it is sometimes convenient to transform it to another game with payoff matrix A − (A + A ⊺ )/N , where A ⊺ means the transpose of A 58,59 . Then, the expression in Eq. (11) can be rewritten as follows: In terms of these transformed payoffs, the difference between π p (i) and π q (i) in Eq. (8) is expressed as (30) γ p,q = D(p, q, f CC ) D(p, q, f 1 ) + D(p, q, f CD ) D(p, q, f 1 ) .
π q,q −π p,q . At each time step, a mutant strategy is randomly selected from the set of deterministic memory-1 strategies ( Table 1). The resident strategy is replaced by the mutant with the fixation probability calculated by Eq. (17). Throughout this paper, we use an error rate e = 10 −3 and selection strengths σ in = σ out = 10 as our baseline values. We conducted the simulations for 10 6 MC steps while discarding the initial 10 5 steps, and took the average over five independent runs to obtain a stationary distribution over the strategy space. While it is possible to obtain the exact stationary distribution of the Markov chains, this method is more vulnerable to underflow. Instead, we found MC simulations to be more reliable, and our simulations are long enough to ignore the statistical fluctuations. For numerical calculation of the fixation probability and the fixation times, we use algorithms based on a previous work 60 .
For the MC simulations for the scenario with a partial separation of time scales, we proceed as follows. First, we prepare a set of randomly selected strategies of size M as an initial state. For each time step, a group i is randomly selected out of the M groups. A player in this group then mutates with probability r. Otherwise, out-group imitation occurs. In case of a mutation, the mutant strategy is randomly selected from the memory-1 strategy space, and the group i is taken over by the mutant with probability Eq. (11). In case of an out-group imitation, a role model is randomly selected from the groups other than i. The strategy of group-i is then replaced with probability f out p→q ρ q,p since a single q player appears in i with f out p→q and the player succeeds in taking over the group with ρ q,p . We define M Monte Carlo steps as one MC sweep, and the simulations are conducted for 1 million sweeps. Again, the initial 10% of data are discarded as the initialization period. The results are averaged over five independent runs. To make the typical time scale comparable between different b, the selection strengths are set to σ in = σ out = 30/(b − 1) . We used OACIS to manage simulation results 61 .
Results on fixation times. One can also calculate the unconditional and conditional fixation times in a group-structured population 9,60 . To simplify the analysis, we assume in the following that the intra-group dynamics are fast enough such that each group can be considered as homogeneous. In other words, we study the dynamics triggered by out-group imitation events and we take 1/µ (the number of the inter-group imitation events) as the unit of time. First, we consider the unconditional fixation time for the inter-group dynamics, which is defined as the average time until the absorbing states (either fixation or extinction) started from the state where there is a single group of mutants. As a preparation, the probability that p succeeds in taking over the population from the state where the number of p groups is i is Using this, the unconditional fixation time is given as The conditional fixation time, which is defined as the expected time until the fixation of p starting from a single p-group conditioned that p succeeds in taking over the population, is This fixation time can be exceedingly long. For instance, consider the dynamics between AllC and AllD. The intra-group dynamics favor AllD while the inter-group dynamics favor AllC. When there are AllC groups and AllD groups, members in AllD groups tend to consider AllC groups when they engage in out-group imitation. However, the AllC player who newly appears in the AllD group immediately disappears again due to the intragroup dynamics. Thus, AllC fails to spread in AllD groups again and again. In other words, both Q + i and Q − i can be quite small even if η ( ≡ Q − i /Q + i ) remains finite. As a result, the population configuration remains the same for a long time, and the fixation times increase dramatically (unless the selection strengths are sufficiently weak).